Raspberry pi Troubleshooting
Here I document problems I have had with Raspberry pi's and any solutions to them I may have found, in the hope it will be of help to others. As of December 2023 there is only one item here because to be honest, I have found all my pi's to be very reliable. I have had virtually no problems with the pi's themselves in my projects over the last 10 years.
Raspberry pi Gen 1: No USB or Ethernet
While working on my Embedded Gentoo article, after constant power cycling my raspberry pi USB and ethernet stopped working. The pi itself boots, the HDMI works as does the GPIO, and the OS loads fine. After research online the main theory for the failure was poor solder connections on the X1 crystal (unfortunately a common issue with modern lead-free soldered components). I re-soldered the X1 crystal and success! It worked again.
I thought that would be the end of it, the pi worked for about two days, but then it died again. I tried re-soldering the crystal but it no longer helped. Generally further research online was not encouraging, if re-soldering the crystal didn't work the general consensus was that it cannot be resurrected.
I might have come to the same conclusion and given up, if it was not for the fact that after configuring the serial console I was able to look into the kernel logs, where I noticed that while USB and Ethernet were not working, the kernel did still detect the controller:
[ 3.452074] dwc_otg 20980000.usb: DWC OTG Controller [ 3.454804] dwc_otg 20980000.usb: new USB bus registered, assigned bus number 1 [ 3.459816] dwc_otg 20980000.usb: irq 56, io mem 0x00000000 [ 3.462464] Init: Port Power? op_state=1 [ 3.465027] Init: Power Port (0) [ 3.467865] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01 [ 3.472940] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.477945] usb usb1: Product: DWC OTG Controller [ 3.480454] usb usb1: Manufacturer: Linux 6.1.69+ dwc_otg_hcd [ 3.483055] usb usb1: SerialNumber: 20980000.usb [ 3.486829] hub 1-0:1.0: USB hub found [ 3.489565] hub 1-0:1.0: 1 port detected
This was interesting to me, as it looked like USB still worked, but not the ports nor the Ethernet. From memory I know that the pi has a USB hub internally, which connects the SoC's USB channel to two USB ports and a USB<->Ethernet device. This made me think that there may well be some hope in resurrecting the pi's usefulness.
However in order to achieve this I will need to do deeper debugging, which will entail digging further into the pi's design. In another example of why I like the pi ecosystem, they actually provide full schematics for the Raspberry pi's making my life much easier! Here is the schematic for my pi.
Looking into it, I found out that the ethernet chip (a "LAN9512", datasheet here) is the USB hub itself. This can be seen in the following section of the schematic:
The chip is clocked using its own crystal separate to the rest of the pi, as shown here:
The crystal is coupled with two capacitors and a resistor in a standard feedback oscillator circuit, nothing out of the ordinary there. It does however help us understand why re-soldering X1 has been known to cure this issue before, as that crystal can stop working without the pi becoming dead, just the loss of USB and Ethernet.
Having checked the crystal solder points, I then did continuity tests between the capacitors and the resistor and it all seems fine. This means there is no simple break in the circuit to indicate failure. Indeed the fact the pi was not under any mechanical stress when it failed makes this unlikely. At this point I don't know exactly why the circuit has failed. It is possible that the crystal itself has died, or possibly the chip.
Reading deeper through the schematic for the USB system on the pi, I found something that made little sense to me. The USB channels as designed actually seem to short out.
If you look on the schematic for the LAN chip USB layout you can see the USB_DM and USB_DP from the SoC go to USB_DM0 and USB_DP0 on the LAN9512, as expected. Then the LAN9512 internal hub outputs two USB channels to the two USB ports, USB_DM2/DP2 and USB_DM3/DP3. This all makes sense. However first thing that confused me was that USB_DM/DP also went directly to USB port B. This means that USB_DM/DP also connected directly to USB_DM3/DP3.
I initially thought the resistors were there perhaps to handle some oddity in chip oscillations or power fluctuations between the channels, however upon closer inspection R36 and R37 were "zero-ohm" resistors. This means they are actually just plain contact bridges (on SMT PCB's, it is usually easier to use zero-ohm bridges rather than program the machine to bridge it using solder). This means that USB_DM/DP, USB_DM0/DP0 and USB_DM3/DP3 were all connected together.
Now USB is not a shared bus architecture (like e.g. I2C is for example) but a spoke and hub architecture. You can't just connect devices and controllers to the same data line pairs expect it all to work. At best one device will work, at worst none of them will.
So looking at this schematic, the USB ports and Ethernet should never have worked. Yet out there are millions of pi's that happily do work. Something was amiss. Therefore I had to look at the PCB itself and see if the schematic is actually correct (it is possible that later errata changed the layout vs the schematic).
So I looked at the base of my pi, and I found R36 and R37, near the ethernet port (makes sense they would be near the LAN chip):
To my surprise there were no resistors there, which means R36 and R37 on the schematic were not populated. This explained to me why the USB/LAN system works on the pi's, but not why they designed the circuit this way. Then it hit me: the design was made to be able to bypass the internal Ethernet chip and hub and give a direct output from the SoC to the USB port B. I suspect this was designed to allow testing of the SoC USB controller by bypassing the Ethernet chip completely, and bridging the controller straight to port B.
This is good news for me, because it means if I cannot get the LAN chip to work, I may well be able to bypass it completely and make the pi usable with an external hub and USB peripherals. All I would have to do is disable the LAN chip and then bridge R36 and R37.
Before I decide to do that however, I would like to try to get the chip to work again. Below is the list of things I have checked:
Power
The schematic shows that the LAN chip runs on 3.3V, so I checked was to make sure we were getting the right voltage to the chip by measuring the two feeds from the capacitors. This was fine.
I also see a secondary 1.8V power feed to the chip, so I checked this. This was fine as well
Those are the only power feeds to the chip, and both seem ok so it is unlikely to be power related.
Oscillator
For this I had to get my oscilloscope out to see if the oscillator is oscillating. There is only one oscillator on the LAN chip, the master clock (X1) which the datasheet says runs at 25MHz.
First thing I did was measure the power supply positive and negative, because I expect there to be some oscillations/noise on the power lines just from everything running on the pi PCB. I got a reading of between 200KHz and 310KHz variable, so any similar reading I will discount as noise.
We measured the X1 oscillator at the oscillators pins (3 and 4), which were easiest to reach. It was oscillating at 2.5MHz, which seems incorrect. The crystal is marked as 25MHz, so it is only oscillating at 1/10th the expected rate. Also while probing I got USB errors, so at least the controller seems to be active:
[ 5.751987] Indeed it is in host mode hprt0 = 00021501 [ 5.961881] usb 1-1: new high-speed USB device number 2 using dwc_otg [ 5.970733] Indeed it is in host mode hprt0 = 00001101 [ 6.181862] usb 1-1: device descriptor read/64, error -71 [ 6.302348] Indeed it is in host mode hprt0 = 00001101 [ 6.511880] usb 1-1: device descriptor read/64, error -71 [ 6.632022] Indeed it is in host mode hprt0 = 00001101 [ 6.841875] usb 1-1: new high-speed USB device number 3 using dwc_otg [ 6.850662] Indeed it is in host mode hprt0 = 00001101 [ 7.061865] usb 1-1: device descriptor read/64, error -71 [ 7.182236] Indeed it is in host mode hprt0 = 00001101 [ 7.391882] usb 1-1: device descriptor read/64, error -71 [ 7.511972] usb usb1-port1: attempt power cycle
Of course being an oscillator, I can expect that measuring the device while active will shift the frequency. After all I am changing the parameters of the system by attaching the probes. However the fact it is measured at exactly 1/10th the supposed oscillation seems "too neat" an occurance to be random so I wanted to confirm.
To help confirm, I measured the only other known working oscillator in the system, X2, which is the clock for the SoC itself. Measuring across C86 I got between 1.8 and 2.0MHz, again around 1/10th the 19.2MHz the crystal should be working at. Also the SoC would hard lockup when I did this, forcing the pi to reset before anything works again. This tells me that rather than the frequency being incorrectly measured, the frequency itself is shifting. It also leads credence to the hypothesis that my probes are be the main cause of the frequency shift. Thus I can infer that my LAN chip oscillator is working correctly at 25MHz when not being disturbed by probes.
Conclusion
Unfortunately at this point I've run out of things to check. I spent some more time tracing lines, testing and measuring at different points in the circuit with no breakthrough. Unless the LAN chip itself gave up the ghost I am not sure what has failed here.
However as mentioned earlier in this post, during my research I did find out about the ability to bypass the LAN chip completely. Therefore in an effort to get at least one USB port working, I decided to bridge R36 and R37 to bypass the LAN chip. I did not do anything else to the chip to disable it, as I suspect given there is only one master on the bus and the chip is a USB hub the only result would be that the hub ceases to work and you get a direct USB link to USB port B.
So I bridged the resistors, powered on the pi, and tried plugging in a USB keyboard. The results were promising!
root@Embedded_Gentoo $ [ 46.001966] Indeed it is in host mode hprt0 = 00041901 [ 46.211870] usb 1-1: new low-speed USB device number 3 using dwc_otg [ 46.221311] Indeed it is in host mode hprt0 = 00041901 [ 46.468841] usb 1-1: New USB device found, idVendor=413c, idProduct=2113, bcdDevice= 1.08 [ 46.480833] usb 1-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0 [ 46.491714] usb 1-1: Product: Dell KB216 Wired Keyboard [ 46.506280] input: Dell KB216 Wired Keyboard as /devices/platform/soc/20980000.usb/usb1/ 1-1/1-1:1.0/0003:413C:2113.0003/input/input3 [ 46.593090] hid-generic 0003:413C:2113.0003: input,hidraw0: USB HID v1.11 Keyboard [Dell KB216 Wired Keyboard] on usb-20980000.usb-1/input0 [ 46.617108] input: Dell KB216 Wired Keyboard System Control as /devices/platform/soc/ 20980000.usb/usb1/1-1/1-1:1.1/0003:413C:2113.0004/input/input4 [ 46.702561] input: Dell KB216 Wired Keyboard Consumer Control as /devices/pla tform/soc/20980000.usb/usb1/1-1/1-1:1.1/0003:413C:2113.0004/input/input5 [ 46.720779] hid-generic 0003:413C:2113.0004: input,hidraw1: USB HID v1.11 Device [Dell KB216 Wired Keyboard] on usb-20980000.usb-1/input1
The keyboard was detected, and I was able to use it just fine! Next I decided to plug in a USB pen drive as that is a high-speed device, to make sure there are no issues with higher clocked devices in this configuration:
[ 364.462004] Indeed it is in host mode hprt0 = 00021501 [ 364.671907] usb 1-1: new high-speed USB device number 4 using dwc_otg [ 364.680998] Indeed it is in host mode hprt0 = 00001101 [ 364.926537] usb 1-1: New USB device found, idVendor=0951, idProduct=1600, bcdDevice= 1.00 [ 364.938668] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 364.949726] usb 1-1: Product: DataTraveler II [ 364.956214] usb 1-1: Manufacturer: Kingston [ 364.970625] usb-storage 1-1:1.0: USB Mass Storage device detected [ 364.979964] scsi host0: usb-storage 1-1:1.0 [ 366.002855] scsi 0:0:0:0: Direct-Access Kingston DataTraveler II PMAP PQ: 0 ANSI: 0 CCS [ 366.178718] sd 0:0:0:0: [sda] 502784 512-byte logical blocks: (257 MB/246 MiB) [ 366.190648] sd 0:0:0:0: [sda] Write Protect is off [ 366.198133] sd 0:0:0:0: [sda] No Caching mode page found [ 366.205576] sd 0:0:0:0: [sda] Assuming drive cache: write through [ 366.218568] sda: sda1 [ 366.223797] sd 0:0:0:0: [sda] Attached SCSI removable disk
Again it was detected just fine! I then tried mounting it and copying data to/from it:
root@Embedded_Gentoo $ fdisk -l /dev/sda Disk /dev/sda: 245 MB, 257425408 bytes, 502784 sectors 3222 cylinders, 6 heads, 26 sectors/track Units: sectors of 1 * 512 = 512 bytes Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type /dev/sda1 4,1,3 1013,5,26 2048 502783 500736 244M 83 Linux root@Embedded_Gentoo $ mount /dev/sda1 /tmp root@Embedded_Gentoo $ ls /tmp ALL0000 ALL0003 ALL0006 ALL0009 ALL0012 ALL0001 ALL0004 ALL0007 ALL0010 ALL0013 ALL0002 ALL0005 ALL0008 ALL0011 TEK0000.BMP root@Embedded_Gentoo $ dd if=/dev/zero of=/tmp/test.img bs=1024 count=30000 30000+0 records in 30000+0 records out 30720000 bytes (29.3MB) copied, 1.900713 seconds, 15.4MB/s root@Embedded_Gentoo $ dd if=/tmp/test.img of=/dev/zero bs=1024 30000+0 records in 30000+0 records out 30720000 bytes (29.3MB) copied, 0.362647 seconds, 80.8MB/s
I retried the "dd" copying to/fro the device a few times, and no problems what so ever happened. The speeds are what is expected and no errors from the kernel have occurred.
So in the end I consider this a success. I have managed to regain USB functionality in my pi. While it would have been nice to get the internal LAN chip and USB hub working again, getting one USB port to work is a good second place. A Raspberry pi without any USB or LAN output is nowhere near as useful as one that has it, and now I can connect external devices, including hubs and ethernet ports to regain lost functionality. This pi will live on!
